x
f
θ
(x) = b + V sigmoid(c + W x),
sigmoid(a) = 1/(1 + e
−a
)
x ∈ R
n
i
h = sigmoid(c + W x) n
h
θ = (b, c, V , W ) θ
b ∈ R
n
o
n
o
c ∈ R
n
h
h
V ∈ R
n
o
×n
h
W ∈ R
n
h
×n
i
L(
ˆ
y, y) = ||
ˆ
y − y||
2
ˆ
y
E[Y | x] L
2
||ω||
2
= (
ij
W
2
ij
+
ki
V
2
ki
) ω
W V L
2
λ
J(θ) = λ||ω||
2
+
1
n
n
t=1
||y
(t)
− (b + V sigmoid(c + W x
(t)
))||
2
.
(x
(t)
, y
(t)
) t
θ
ω ← ω −
2λω + ∇
ω
L(f
θ
(x
(t)
), y
(t)
)
β ← β −∇
β
L(f
θ
(x
(t)
), y
(t)
),
β = (b, c) ω = (W , V )
t
n
h
k
= tanh(b
k
+ W
k
h
k−1
)
h
0
= x h
k
k > 0
k W
k
b
k
f
θ
(x)
tanh
max(0, b + w · x) max
i
(b
i
+ W
:,i
· x)
a = b + W x
h = φ(a) ⇔ h
i
= φ(a
i
) = φ(b
i
+ W
i,:
x).
•
φ(a) = max(0, a)
φ(a) = (a)
+
α
i
a < 0 h
i
= φ(a, α
i
) = max(0, a) + α
i
min(0, a) α
i
α
i
α
i
• φ(a) = tanh(a)
• φ(a) = 1/(1 + e
−a
)
• φ(a) = softmax(a) =
e
a
i
/
j
e
a
j
i
φ
i
(a) = 1 φ
i
(a) > 0
tanh(x) = 2 × sigmoid(2x) − 1
1
=1
{1, . . . , n}
n −1
p
i
= P ( = i | x) P ( = N | x) 1 −
n−1
i=1
P ( = i | x).
n
p = softmax(a) ⇐⇒ p
i
=
e
a
i
j
e
a
j
.
a
a = b+W h a
n
n−1 a
a
i
i
L
NLL
(p, y) = −log p
y
a
y
a
y
y a
i
i = y x
p
y
= 1 x p
i
= i x
a
∂
∂a
k
L
NLL
(p, y) =
∂
∂a
k
(−log p
y
) =
∂
∂a
k
(−a
y
+ log
j
e
a
j
)
= −1
y=k
+
e
a
k
j
e
a
j
= p
k
− 1
y=k
or
∂
∂a
L
NLL
(p, y) = (p −e
y
)
e
y
= [0, . . . , 0, 1, 0, . . . , 0] y
x a a
p = E
y
[e
y
| x]
p
i
y = i
p
i
P (y = i | x)
e
y
y y
logp(y = i | x) a p(y = i | x)
i = i
j = i
∂
∂a
j
L
NLL
(p, y) = p
j
.
= j p
j
≈ 0
j p
j
≈ 1
a
j
y p
y
≈ 0
a
y
a
j
a
j
a
j
a
j
p
j
≈ 1 a
j
a
j
p
y
≈ 0 a
y
a
y
p
y
− 1
y=y
= p
y
− 1 < 0
a
j
j = y
p
j
> 0
y = e
i
= [0, . . . , 0, 1, 0, . . . , 0]
= i y
i
= 1 y
j
= 0, ∀j = i
p = softmax(a) a
a = b + W h h
L
2
(p(a), y) = ||p(a) −y||
2
a
∂
∂a
i
L
2
(p(a), y) =
∂L
2
(p(a), y)
∂p(a)
∂p(a)
∂a
i
=
j
2(p
j
(a) −y
j
)p
j
(1
i=j
− p
i
).
= i
p
y
= p
i
≈ 0 a
y
∂
∂a
y
L
2
(p(a), y) ≈ 0
softmax(a) = softmax(a + b).
softmax(a) = softmax(a −max
i
a
i
).
a
a
i
p(y | ω) ω
p(y | x) ω
x
p(y | ω = f
θ
(x))
f
θ
(x) x y
f
θ
(x)
y ω = f
θ
(x)
y
y
ω
y
x
ω
L(x, y) = −log p(y | ω = f
θ
(x))
y
y y x
y x
x x
σ
2
y x x
y f
θ
(x)
σ
σ
2
←
1
n
n
i=1
(y
(t)
− f
θ
(x
(t)
))
2
(t)
t (x
(t)
, y
(t)
)
y d σ
2
σ
σ
2
←
1
nd
n
i=1
||y
(t)
− f
θ
(x
(t)
)||
2
.
σ
2
i
σ
2
i
←
1
n
n
i=1
(y
(t)
i
− f
θ,i
(x
(t)
))
2
.
Σ ←
1
n
n
i=1
(y
(t)
i
− f
θ,i
(x
(t)
))(y
(t)
i
− f
θ,i
(x
(t)
))
Σ(x) x
Σ(x)
σ
i
(x) = softplus(g
θ
(x)).
g
θ
(x) x
σ x
ω
σ
i
(x) = softplus(ω
i
)
Σ(x) = B(x)B
(x)
B
O(d
3
)
Σ(x)
B(x)
ω = p( = 1 | x)
ω
n
p(y | x) =
n
i=1
p( = i | x)N(y | µ
i
(x), Σ
i
(x)).
p( = i | x) µ
i
(x) Σ
i
(x)
µ σ
2
∼ N(µ, σ
2
).
µ σ
2
η ∼ N(0, 1)
z = µ + ση
η
µ σ
η
J(z)
µ = f(x; θ) σ = g(x; θ)
∇
θ
J(z)
z p( | ω) ω
z ∼ p( | ω)
z = f(ω, η)
η
ω
z
f
z ω
z f
ω
η
f ω
J(z) b(ω)
z
E
p(z)
∂ log p(z)
∂ω
=
z
p(z)
∂ log p(z)
∂ω
=
z
p(z)
∂ω
=
∂
∂ω
z
p(z) =
∂
∂ω
1 = 0,
E
p(z)
(J(z) − b(ω))
∂ log p(z)
∂ω
= E
p(z)
J(z)
∂ log p(z)
∂ω
− b(ω)E
p(z)
∂ log p(z)
∂ω
= E
p(z)
J(z)
∂ log p(z)
∂ω
.
b(ω) (J(z)−
b(ω))
log p(z)
∂ω
p(z) b(ω)
b
∗
i
(ω) ω
i
ω
b
∗
i
(ω) =
E
p(z)
J(z)
∂ log p(z)
∂ω
i
2
E
p(z)
∂ log p(z)
∂ω
i
2
.
ω
i
(J(z) − b
i
(ω))
∂ log p(z)
∂ω
i
b
i
(ω) b
∗
i
(ω)
ω
E
p(z)
[J(z)
∂ log p(z)
∂ω
i
2
] E
p(z)
∂ log p(z)
∂ω
i
2
ω
∂ log p(z)
∂ω
i
2
∂ log p(z)
∂ω
i
2
z
p(z) ω
i ω J(z)
b(ω) ≈ E
p(z)
[J(z)]
b(ω) (J(z)−b(ω))
ω
ω
z J(z)
z
R
n
v ∈ {0, 1}
n
2
2
n
2
n
O(2
n
)
d
d
n
0.0 0.2 0.4 0.6 0.8 1.0
Number of parameters
1e8
91
92
93
94
95
96
97
Test accuracy (%)
Effect of Number of Parameters
3, convolutional
3, fully connected
5
11, convolutional
n
k k
k
